Content-based detection and three dimensional geometric reconstruction of objects in image and video data

ABSTRACT

Systems, computer program products, and techniques for detecting and/or reconstructing objects depicted in digital image data within a three-dimensional space are disclosed, according to various exemplary embodiments. The inventive concepts uniquely utilize internal features to accomplish reconstruction, thereby avoiding reliance on reconstructing objects based on information derived from location of edges. The inventive concepts thus provide an improvement over conventional object reconstruction since objects may be detected and/or reconstructed even when edges are obscured or not depicted in the digital image data. In one aspect, reconstructing an object depicted in a digital image includes using a processor to: detect a plurality of identifying features of the object, where the identifying features are located internally with respect to the object; and reconstruct the digital image of the object within a three dimensional coordinate space based at least in part on some or all of the identifying features.

FIELD OF INVENTION

The present invention relates to digital image data capture andprocessing, and more particularly to detecting objects depicted in imageand/or video data based on internally-represented features (content) asopposed to edges. The present invention also relates to reconstructingobjects in a three-dimensional coordinate space so as to rectify imageartifacts caused by distortional effects inherent to capturing imageand/or video data using a camera.

BACKGROUND OF THE INVENTION

Digital images having depicted therein a document such as a letter, acheck, a bill, an invoice, a credit card, a driver license, a passport,a social security card, etc. have conventionally been captured andprocessed using a scanner or multifunction peripheral coupled to acomputer workstation such as a laptop or desktop computer. Methods andsystems capable of performing such capture and processing are well knownin the art and well adapted to the tasks for which they are employed.

However, in an era where day-to-day activities, computing, and businessare increasingly performed using mobile devices, it would be greatlybeneficial to provide analogous document capture and processing systemsand methods for deployment and use on mobile platforms, such as smartphones, digital cameras, tablet computers, etc.

A major challenge in transitioning conventional document capture andprocessing techniques is the limited processing power and imageresolution achievable using hardware currently available in mobiledevices. These limitations present a significant challenge because it isimpossible or impractical to process images captured at resolutionstypically much lower than achievable by a conventional scanner. As aresult, conventional scanner-based processing algorithms typicallyperform poorly on digital images captured using a mobile device.

In addition, the limited processing and memory available on mobiledevices makes conventional image processing algorithms employed forscanners prohibitively expensive in terms of computational cost.Attempting to process a conventional scanner-based image processingalgorithm takes far too much time to be a practical application onmodern mobile platforms.

A still further challenge is presented by the nature of mobile capturecomponents (e.g. cameras on mobile phones, tablets, etc.). Whereconventional scanners are capable of faithfully representing thephysical document in a digital image, critically maintaining aspectratio, dimensions, and shape of the physical document in the digitalimage, mobile capture components are frequently incapable of producingsuch results.

Specifically, images of documents captured by a camera present a newline of processing issues not encountered when dealing with imagescaptured by a scanner. This is in part due to the inherent differencesin the way the document image is acquired, as well as the way thedevices are constructed. The way that some scanners work is to use atransport mechanism that creates a relative movement between paper and alinear array of sensors. These sensors create pixel values of thedocument as it moves by, and the sequence of these captured pixel valuesforms an image. Accordingly, there is generally a horizontal or verticalconsistency up to the noise in the sensor itself, and it is the samesensor that provides all the pixels in the line.

In contrast, cameras have many more sensors in a nonlinear array, e.g.,typically arranged in a rectangle. Thus, all of these individual sensorsare independent, and render image data that is not typically ofhorizontal or vertical consistency. In addition, cameras introduce aprojective effect that is a function of the angle at which the pictureis taken. For example, with a linear array like in a scanner, even ifthe transport of the paper is not perfectly orthogonal to the alignmentof sensors and some skew is introduced, there is no projective effectlike in a camera. Additionally, with camera capture, nonlineardistortions may be introduced because of the camera optics.

Distortions and blur are particularly challenging when attempting todetect objects represented in video data, as the camera typically moveswith respect to the object during the capture operation, and video dataare typically characterized by a relatively low resolution compared tostill images captured using a mobile device. Moreover, the motion of thecamera may be erratic and occur within three dimensions, meaning thehorizontal and/or vertical consistency associated with linear motion ina conventional scanner is not present in video data captured usingmobile devices. Accordingly, reconstructing an object to correct fordistortions, e.g. due to changing camera angle and/or position, within athree-dimensional space is a significant challenge.

Further still, as mobile applications increasingly rely on or leverageimage data to provide useful services to customers, e.g. mobile banking,shopping, applying for services such as loans, opening accounts,authenticating identity, acquiring or renewing licenses, etc., capturingrelevant information within image data is a desirable capability.However, often the detection of objects within the mobile image data isa challenging task, particularly where the object's edges may bemissing, obscured, etc. within the captured image/video data. Sinceconventional detection techniques rely on detecting objects by locatingedges of the object (i.e. boundaries between the object, typicallyreferred to as the image “foreground” and the background of the image orvideo), missing or obscured object edges present an additional obstacleto consistent and accurate object detection.

In view of the challenges presented above, it would be beneficial toprovide an image capture and processing algorithm and applicationsthereof that compensate for and/or correct problems associated withusing a mobile device to capture and/or detect objects within imageand/or video data, and reconstruct such objects within athree-dimensional coordinate space.

SUMMARY OF THE INVENTION

According to one embodiment, a computer program product forreconstructing an object depicted in a digital image includes a computerreadable medium having stored thereon computer readable programinstructions configured to cause a hardware processor, upon executionthereof, to: detect a plurality of identifying features of the object,wherein the plurality of identifying features are located internallywith respect to the object; and reconstruct the digital image of theobject within a three dimensional coordinate space based at least inpart on some or all of the plurality of identifying features.

According to another embodiment, a computer program product fordetecting a financial instrument depicted in a digital image, comprisinga computer readable medium having stored thereon computer readableprogram instructions configured to cause a hardware processor, uponexecution thereof, to: detect, using the hardware processor, a pluralityof identifying features of the financial instrument, wherein theplurality of identifying features are located internally with respect tothe financial instrument; project, using the hardware processor, alocation of one or more edges of the financial instrument based at leastin part on the plurality of identifying features; and output, at leastin part based on the projected location of the one or more edges of thefinancial instrument to a display of a computer, and/or a memory.Notably, the financial instrument is a check.

According to yet another embodiment, a computer program product fordetecting a object depicted in a digital image, comprising a computerreadable medium having stored thereon computer readable programinstructions configured to cause a hardware processor, upon executionthereof, to: detect, using the hardware processor, a plurality ofidentifying features of the object, wherein the plurality of identifyingfeatures are located internally with respect to the object; project,using the hardware processor, a location of one or more edges of theobject based at least in part on the plurality of identifying features;and output, at least in part based on the projected location of the oneor more edges of the object to a display of a computer, and/or a memory.

Other aspects and embodiments of the invention will be appreciated basedon reviewing the following descriptions in full detail.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network architecture, in accordance with oneembodiment.

FIG. 2 shows a representative hardware environment that may beassociated with the servers and/or clients of FIG. 1, in accordance withone embodiment.

FIG. 3A is a digital image of a document including a plurality ofdesignated feature zones, according to one embodiment.

FIG. 3B is a digital image of a document including a plurality ofdesignated identifying features, according to one embodiment.

FIG. 3C is a digital image of a document including an extended set ofthe plurality of designated identifying features, according to anotherembodiment.

FIG. 4A depicts a mapping between matching distinctive features of areference image and test image of a driver license, according to oneembodiment.

FIG. 4B depicts a mapping between matching distinctive features of areference image and test image of a driver license, according to anotherembodiment where the test and reference images depict the driver licenseat different rotational orientations.

FIG. 4C depicts a mapping between matching distinctive features of areference image and test image of a credit card, according to oneembodiment.

FIG. 5 is a simplified schematic of a credit card having edges thereofprojected based on internal features of the credit card, according toone embodiment.

FIG. 6A is a simplified schematic showing a coordinate system formeasuring capture angle, according to one embodiment.

FIG. 6B depicts an exemplary schematic of a rectangular object capturedusing a capture angle normal to the object, according to one embodiment.

FIG. 6C depicts an exemplary schematic of a rectangular object capturedusing a capture angle slightly skewed with respect to the object,according to one embodiment.

FIG. 6D depicts an exemplary schematic of a rectangular object capturedusing a capture angle significantly skewed with respect to the object,according to one embodiment.

FIG. 7 is a flowchart of a method for detecting objects depicted indigital images based on internal features of the object, according toone embodiment.

FIG. 8 is a flowchart of a method for reconstructing objects depicted indigital images based on internal features of the object, according toone embodiment.

DETAILED DESCRIPTION

The following description is intended to illustrate the generalprinciples of the present invention and is not meant to limit theinventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless otherwise specified.

The present application refers to image processing. In particular, thepresent application discloses systems, methods, and computer programproducts configured to detect and reconstruct objects depicted indigital images from a non-rectangular shape to a substantiallyrectangular shape, or preferably a rectangular shape. Even morepreferably, this is accomplished based on evaluating the internalfeatures of the object(s) rather than detecting object edges andreconstructing a particular shape based on edge contours.

According to one embodiment, a computer-implemented method ofreconstructing an object depicted in a digital image includes: detectinga plurality of identifying features of the object, wherein the pluralityof identifying features are located internally with respect to theobject; and reconstructing the digital image of the object within athree dimensional coordinate space based at least in part on some or allof the plurality of identifying features.

According to another embodiment, a computer program product forreconstructing an object depicted in a digital image includes a computerreadable medium having stored thereon computer readable programinstructions configured to cause a processor, upon execution thereof,to: detect, using the processor, a plurality of identifying features ofthe object, wherein the plurality of identifying features are locatedinternally with respect to the object; and reconstruct, using theprocessor, the digital image of the object within a three dimensionalcoordinate space based at least in part on some or all of the pluralityof identifying features.

According to yet another embodiment, a system for reconstructing anobject depicted in a digital image includes a processor and logicembodied with and/or executable by the processor. The logic isconfigured to cause the processor, upon execution thereof, to: detect aplurality of identifying features of the object, wherein the pluralityof identifying features are located internally with respect to theobject; and reconstruct the digital image of the object within a threedimensional coordinate space based at least in part on some or all ofthe plurality of identifying features.

The following definitions will be useful in understanding the inventiveconcepts described herein, according to various embodiments. Thefollowing definitions are to be considered exemplary, and are offeredfor purposes of illustration to provide additional clarity to thepresent disclosures, but should not be deemed limiting on the scope ofthe inventive concepts disclosed herein.

As referred to henceforth, a “quadrilateral” is a four-sided figurewhere (1) each side is linear, and (2) adjacent sides form vertices atthe intersection thereof. Exemplary quadrilaterals are depicted in FIGS.6C and 6D below, according to two illustrative embodiments.

A “parallelogram” is a special type of quadrilateral, i.e. a four-sidedfigure where (1) each side is linear, (2) opposite sides are parallel,and (3) adjacent sides are not necessarily perpendicular, such thatvertices at the intersection of adjacent sides form angles having valuesthat are not necessarily 90°.

A “rectangle” or “rectangular shape” is a special type of quadrilateral,which is defined as a four-sided figure, where (1) each side is linear,(2) opposite sides are parallel, and (3) adjacent sides areperpendicular, such that an interior angle formed at the vertex betweeneach pair of adjacent sides is a right-angle, i.e. a 90° angle. Anexemplary rectangle is depicted in FIG. 6B, according to oneillustrative embodiment.

Moreover, as referred-to herein “rectangles” and “rectangular shapes”are considered to include “substantially rectangular shapes”, which aredefined as a four-sided shape where (1) each side is predominantlylinear (e.g. at least 90%, 95%, or 99% of each side's length, in variousembodiments, is characterized by a first-order polynomial (such asy=mx+b), (2) each pair of adjacent sides form an interior angle having avalue θ, where θ is approximately 90° (e.g. θ satisfies therelationship: 85°≤θ≤95°)) at either (a) a vertex between two adjacentsides, (b) a vertex between a projection of the predominantly linearportion of one side and an adjacent side, or (c) a vertex between aprojection of the predominantly linear portion of one side and aprojection of the predominantly linear portion of an adjacent side.

A “non-rectangular shape” as referred to herein includes any shape thatis not either a “rectangular shape” or a “substantially rectangularshape” as defined above. In preferred embodiments, a “non-rectangularshape” is a “tetragon,” which as referred to herein is a four-sidedfigure, where: (1) each side is characterized in whole or in part by anequation selected from a chosen class of functions (e.g. selected from aclass of polynomials preferably ranging from zeroth order to fifthorder, more preferably first order to third order polynomials, and evenmore preferably first order to second order polynomials), and (2)adjacent sides of the figure form vertices at the intersection thereof.

Images (e.g. pictures, figures, graphical schematics, single frames ofmovies, videos, films, clips, etc.) are preferably digital imagescaptured by cameras, especially cameras of mobile devices. As understoodherein, a mobile device is any device capable of receiving data withouthaving power supplied via a physical connection (e.g. wire, cord, cable,etc.) and capable of receiving data without a physical data connection(e.g. wire, cord, cable, etc.). Mobile devices within the scope of thepresent disclosures include exemplary devices such as a mobiletelephone, smartphone, tablet, personal digital assistant, iPod®, iPad®,BLACKBERRY® device, etc.

However, as it will become apparent from the descriptions of variousfunctionalities, the presently disclosed mobile image processingalgorithms can be applied, sometimes with certain modifications, toimages coming from scanners and multifunction peripherals (MFPs).Similarly, images processed using the presently disclosed processingalgorithms may be further processed using conventional scannerprocessing algorithms, in some approaches.

Of course, the various embodiments set forth herein may be implementedutilizing hardware, software, or any desired combination thereof. Forthat matter, any type of logic may be utilized which is capable ofimplementing the various functionality set forth herein.

One benefit of using a mobile device is that with a data plan, imageprocessing and information processing based on captured images can bedone in a much more convenient, streamlined and integrated way thanprevious methods that relied on presence of a scanner. However, the useof mobile devices as document(s) capture and/or processing devices hasheretofore been considered unfeasible for a variety of reasons.

In one approach, an image may be captured by a camera of a mobiledevice. The term “camera” should be broadly interpreted to include anytype of device capable of capturing an image of a physical objectexternal to the device, such as a piece of paper. The term “camera” doesnot encompass a peripheral scanner or multifunction device. Any type ofcamera may be used. Preferred embodiments may use cameras having ahigher resolution, e.g. 8 MP or more, ideally 12 MP or more. The imagemay be captured in color, grayscale, black and white, or with any otherknown optical effect. The term “image” as referred to herein is meant toencompass any type of data corresponding to the output of the camera,including raw data, processed data, etc.

The description herein is presented to enable any person skilled in theart to make and use the invention and is provided in the context ofparticular applications of the invention and their requirements. Variousmodifications to the disclosed embodiments will be readily apparent tothose skilled in the art and the general principles defined herein maybe applied to other embodiments and applications without departing fromthe spirit and scope of the present invention. Thus, the presentinvention is not intended to be limited to the embodiments shown, but isto be accorded the widest scope consistent with the principles andfeatures disclosed herein.

General Computing and Networking Concepts

In particular, various embodiments of the invention discussed herein areimplemented using the Internet as a means of communicating among aplurality of computer systems. One skilled in the art will recognizethat the present invention is not limited to the use of the Internet asa communication medium and that alternative methods of the invention mayaccommodate the use of a private intranet, a Local Area Network (LAN), aWide Area Network (WAN) or other means of communication. In addition,various combinations of wired, wireless (e.g., radio frequency) andoptical communication links may be utilized.

The program environment in which one embodiment of the invention may beexecuted illustratively incorporates one or more general-purposecomputers or special-purpose devices such hand-held computers. Detailsof such devices (e.g., processor, memory, data storage, input and outputdevices) are well known and are omitted for the sake of clarity.

It should also be understood that the techniques of the presentinvention might be implemented using a variety of technologies. Forexample, the methods described herein may be implemented in softwarerunning on a computer system, or implemented in hardware utilizing oneor more processors and logic (hardware and/or software) for performingoperations of the method, application specific integrated circuits,programmable logic devices such as Field Programmable Gate Arrays(FPGAs), and/or various combinations thereof. In one illustrativeapproach, methods described herein may be implemented by a series ofcomputer-executable instructions residing on a storage medium such as aphysical (e.g., non-transitory) computer-readable medium. In addition,although specific embodiments of the invention may employobject-oriented software programming concepts, the invention is not solimited and is easily adapted to employ other forms of directing theoperation of a computer.

The invention can also be provided in the form of a computer programproduct comprising a computer readable storage or signal medium havingcomputer code thereon, which may be executed by a computing device(e.g., a processor) and/or system. A computer readable storage mediumcan include any medium capable of storing computer code thereon for useby a computing device or system, including optical media such as readonly and writeable CD and DVD, magnetic memory or medium (e.g., harddisk drive, tape), semiconductor memory (e.g., FLASH memory and otherportable memory cards, etc.), firmware encoded in a chip, etc.

A computer readable signal medium is one that does not fit within theaforementioned storage medium class. For example, illustrative computerreadable signal media communicate or otherwise transfer transitorysignals within a system, between systems e.g., via a physical or virtualnetwork, etc.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1, a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the present networkarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. Such user devices 116 mayinclude a desktop computer, laptop computer, hand-held computer, printeror any other type of logic. It should be noted that a user device 111may also be directly coupled to any of the networks, in one embodiment.

A peripheral 120 or series of peripherals 120, e.g. facsimile machines,printers, networked storage units, etc., may be coupled to one or moreof the networks 104, 106, 108. It should be noted that databases,servers, and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesa MAC OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates a MAC OSenvironment, etc. This virtualization and/or emulation may be enhancedthrough the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 108, may represent acluster of systems commonly referred to as a “cloud.” In cloudcomputing, shared resources, such as processing power, peripherals,software, data processing and/or storage, servers, etc., are provided toany system in the cloud, preferably in an on-demand relationship,thereby allowing access and distribution of services across manycomputing systems. Cloud computing typically involves an Internet orother high speed connection (e.g., 4G LTE, fiber optic, etc.) betweenthe systems operating in the cloud, but other techniques of connectingthe systems may also be used.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1, a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the presentarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. Such user devices 116 mayinclude a desktop computer, lap-top computer, hand-held computer,printer or any other type of logic. It should be noted that a userdevice 111 may also be directly coupled to any of the networks, in oneembodiment.

A peripheral 120 or series of peripherals 120, e.g., facsimile machines,printers, networked and/or local storage units or systems, etc., may becoupled to one or more of the networks 104, 106, 108. It should be notedthat databases and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesa MAC OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates a MAC OSenvironment, etc. This virtualization and/or emulation may be enhancedthrough the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 108, may represent acluster of systems commonly referred to as a “cloud.” In cloudcomputing, shared resources, such as processing power, peripherals,software, data processing and/or storage, servers, etc., are provided toany system in the cloud, preferably in an on-demand relationship,thereby allowing access and distribution of services across manycomputing systems. Cloud computing typically involves an Internet orother high speed connection (e.g., 4G LTE, fiber optic, etc.) betweenthe systems operating in the cloud, but other techniques of connectingthe systems may also be used.

FIG. 2 shows a representative hardware environment associated with auser device 116 and/or server 114 of FIG. 1, in accordance with oneembodiment. Such figure illustrates a typical hardware configuration ofa workstation having a central processing unit 210, such as amicroprocessor, and a number of other units interconnected via a systembus 212.

The workstation shown in FIG. 2 includes a Random Access Memory (RAM)214, Read Only Memory (ROM) 216, an I/O adapter 218 for connectingperipheral devices such as disk storage units 220 to the bus 212, a userinterface adapter 222 for connecting a keyboard 224, a mouse 226, aspeaker 228, a microphone 232, and/or other user interface devices suchas a touch screen and a digital camera (not shown) to the bus 212,communication adapter 234 for connecting the workstation to acommunication network 235 (e.g., a data processing network) and adisplay adapter 236 for connecting the bus 212 to a display device 238.

The workstation may have resident thereon an operating system such asthe Microsoft Windows® Operating System (OS), a MAC OS, a UNIX OS, etc.It will be appreciated that a preferred embodiment may also beimplemented on platforms and operating systems other than thosementioned. A preferred embodiment may be written using JAVA, XML, C,and/or C++ language, or other programming languages, along with anobject oriented programming methodology. Object oriented programming(OOP), which has become increasingly used to develop complexapplications, may be used.

Mobile Image Capture

Various embodiments of a Mobile Image Capture and Processing algorithm,as well as several mobile applications configured to facilitate use ofsuch algorithmic processing within the scope of the present disclosuresare described below. It is to be appreciated that each section belowdescribes functionalities that may be employed in any combination withthose disclosed in other sections, including any or up to all thefunctionalities described herein. Moreover, functionalities of theprocessing algorithm embodiments as well as the mobile applicationembodiments may be combined and/or distributed in any manner across avariety of computing resources and/or systems, in several approaches.

An application may be installed on the mobile device, e.g., stored in anonvolatile memory of the device. In one approach, the applicationincludes instructions to perform processing of an image on the mobiledevice. In another approach, the application includes instructions tosend the image to one or more non-mobile devices, e.g. a remote serversuch as a network server, a remote workstation, a cloud computingenvironment, etc. as would be understood by one having ordinary skill inthe art upon reading the present descriptions. In yet another approach,the application may include instructions to decide whether to performsome or all processing on the mobile device and/or send the image to theremote site. Examples of how an image may be processed are presented inmore detail below.

In one embodiment, there may be no difference between the processingthat may be performed on the mobile device and a remote server, otherthan speed of processing, constraints on memory available, etc.Moreover, there may be some or no difference between various userinterfaces presented on a mobile device, e.g. as part of a mobileapplication, and corresponding user interfaces presented on a display incommunication with the non-mobile device.

In other embodiments, a remote server may have higher processing power,more capabilities, more processing algorithms, etc. In yet furtherembodiments, the mobile device may have no image processing capabilityassociated with the application, other than that required to send theimage to the remote server. In yet another embodiment, the remote servermay have no image processing capability relevant to the platformspresented herein, other than that required to receive the processedimage from the remote server. Accordingly, the image may be processedpartially or entirely on the mobile device, and/or partially or entirelyon a remote server, and/or partially or entirely in a cloud, and/orpartially or entirely in any part of the overall architecture inbetween. Moreover, some processing steps may be duplicated on differentdevices.

Which device performs which parts of the processing may be defined by auser, may be predetermined, may be determined on the fly, etc. Moreover,some processing steps may be re-performed, e.g., upon receiving arequest from the user. Accordingly, the raw image data, partiallyprocessed image data, or fully processed image data may be transmittedfrom the mobile device, e.g., using a wireless data network, to a remotesystem. Image data as processed at a remote system may be returned tothe mobile device for output and/or further processing.

In a further approach, the image may be partitioned, and the processingof the various parts may be allocated to various devices, e.g., ½ to themobile device and ½ to the remote server, after which the processedhalves are combined.

In one embodiment, selection of which device performs the processing maybe based at least in part on a relative speed of processing locally onthe mobile device vs. communication with the server.

In one approach, a library of processing functions may be present, andthe application on the mobile device or the application on a remoteserver simply makes calls to this library, and essentially the meaningof the calls defines what kind of processing to perform. The device thenperforms that processing and outputs the processed image, perhaps withsome corresponding metadata.

Any type of image processing known in the art and/or as newly presentedherein may be performed in any combination in various embodiments.

Referring now to illustrative image processing, the camera can beconsidered an area sensor that captures images, where the images mayhave any number of projective effects, and sometimes non-linear effects.The image may be processed to correct for such effects. Moreover, theposition and boundaries of the document(s) in the image may be foundduring the processing, e.g., the boundaries of one or more actual pagesof paper in the background surrounding the page(s). Because of themobile nature of various embodiments, the sheet of paper may be lying onjust about anything. This complicates image analysis in comparison toprocessing images of documents produced using a scanner, because scannerbackground properties are constant and typically known, whereas mobilecapture backgrounds may vary almost infinitely according to the locationof the document and the corresponding surrounding textures captured inthe image background, as well as because of variable lightingconditions.

Accordingly, the non-uniformity of the background of the surface onwhich the piece of paper may be positioned for capture by the camerapresents one challenge, and the non-linear and projective effectspresent additional challenges. Various embodiments overcome thesechallenges, as will soon become apparent.

In one exemplary mode of operation, an application on the mobile devicemay be initiated, e.g., in response to a user request to open theapplication. For example, a user-selection of an icon representing theapplication may be detected.

In some approaches, a user authentication may be requested and/orperformed. For example, a user ID and password, or any otherauthentication information, may be requested and/or received from theuser.

In further approaches, various tasks may be enabled via a graphical userinterface of the application. For example, a list of tasks may bepresented. In such case, a selection of one of the tasks by the user maybe detected, and additional options may be presented to the user, apredefined task may be initiated, the camera may be initiated, etc.

Content-Based Object Detection

An image may be captured by the camera of the mobile device, preferablyupon receiving some type of user input such as detecting a tap on ascreen of the mobile device, depression of a button on the mobiledevice, a voice command, a gesture, etc. Another possible scenario mayinvolve some level of analysis of sequential frames, e.g. from a videostream. Sequential frame analysis may be followed by a switch tocapturing a single high-resolution image frame, which may be triggeredautomatically or by a user, in some approaches. Moreover, the triggermay be based on information received from one or more mobile devicesensors. For example, in one embodiment an accelerometer in or coupledto the mobile device may indicate a stability of the camera, and theapplication may analyze low-resolution video frame(s) for presence of anobject of interest. If an object is detected, the application mayperform a focusing operation and acquire a high-resolution image of thedetected object. Either the low- or high-resolution image may be furtherprocessed, but preferred embodiments utilize the high-resolution imagefor subsequent processing.

In more approaches, switching to single frame mode as discussed abovemay be unnecessary, particularly for smaller objects, in particulardocuments such as business cards, receipts, credit cards, identificationdocuments such as driver licenses and passports, etc. To increaseprocessing rate and reduce consumption of processing resources, objecttype identification may facilitate determining whether or not to switchto single frame mode and/or capture a high-resolution image forprocessing.

As noted above, conventional techniques for detecting objects in imageand/or video data generally rely on detecting the edges of the object,i.e. transitions between the background and foreground (which depictsthe object) of the image or video data. For instance, edges may bedetected based on locating one or more lines (e.g. four linesintersecting to form corners of a substantially rectangular object suchas a document) of pixels characterized by a sharp transition in pixelintensity between the background and foreground.

However, where edges are missing or obscured, the conventional edgedetection approach is not reasonably accurate or consistent in detectingobjects within image and/or video data. Similar challenges exist inimages where the object for which detection is desired is set against acomplex background (e.g. a photograph or environmental scene) sincedetecting sharp transitions in intensity is likely to generate manyfalse positive predictions of the location of the object. Accordingly, anew approach is presented via the inventive concepts disclosed herein,and this inventive approach advantageously does not rely on detectingobject edges to accomplish object detection within the image and/orvideo data.

In particular, the presently disclosed inventive concepts include usingfeatures of the object other than the edges, e.g. content depictedwithin a document, to serve as identifying characteristics from whichobject detection may be accomplished. While the present descriptions setforth several exemplary embodiments of object detection primarily withreference to features of documents, it should be understood that theseconcepts are equally applicable to nearly any type of object, and thetechniques discussed herein may be utilized to detect nearly any type ofobject for which a suitable set of identifying features are presentacross various exemplars of that object type.

Turning now to exemplary embodiments in which the detected object is adocument, e.g. a form, a passport, a driver license, a credit card, abusiness card, a check, a receipt etc., and consistent with the notionthat identifying features should be present across various (preferablyall) exemplars of a particular document type, content that is common todocuments of that type may serve as a suitable identifying feature. Insome approaches, edges of the detected object may be cut off, obscured,or otherwise not identifiable within the image. Indeed, the presentlydisclosed inventive concepts offer the particular advantage thatdetection of objects may be accomplished independent of whether objectedges are identifiable within the image data. Accordingly, the presentlydisclosed inventive concepts effectuate an improvement to systemsconfigured for object recognition/detection within image data.

In some approaches, when the object or document is known to depictparticular content in a particular location, e.g. a barcode, MICRcharacters for a check, MRZ characters on passports and certain types ofidentifying documents, etc., then these reference content may beemployed to facilitate detecting the object within image and/or videodata. In many cases, reference content position and/or content isdefined by some sort of standard. In various embodiments, it isaccordingly advantageous to leverage a priori knowledge regarding thelocation, size, orientation, etc. of reference content within an imageto project the location of document edges based on the reference contentas depicted in the image and/or video data.

However, not all objects include such reference content. Accordingly, inmore embodiments, content such as internal lines, symbols (e.g. smallimages like icons which preferably contain rich texture information, forinstance, for a fingerprint, the ridge pattern, especially, the crosspoints of two lines, etc.), text characters, etc. which appears onsubstantially all documents of a particular type is eligible for use asan identifying feature. According to the present descriptions, suchcontent may also be referred to as “boilerplate content.”

Boilerplate content may be determined manually, e.g. based on a userdefining particular feature zones within a reference image, in someapproaches. For instance, a user may define particular regions such asthose designated in FIG. 3A by dashed-line bounding boxes. In aparticularly preferred approach, the particular regions defined by theuser may include a subset of the regions shown in FIG. 3A, mostpreferably those regions exhibiting a shading within the bounding box(e.g. for a California driver license, state name “CALIFORNIA,”expiration date “EXP,” first name “FN,” last name “LN,” date of birth“DOB,” sex “SEX,” height “HGT,” eye color “EYES,” weight “WGT,” anddocument discriminator “DD” field designators). In various approaches,the feature zones may include boilerplate text, e.g. regions 302 and/ornon-textual identifying features such as logos, lines, intersectinglines, shapes, holograms, designs, drawings, etc. such as represented inregion 304 of FIG. 3A, according to one embodiment.

Upon reading the present descriptions, skilled artisans will appreciatethat the portions of the document obscured by white rectangles areredactions to protect sensitive information, and should not beconsidered feature zones within the scope of the presently disclosedinventive concepts. Indeed, by way of contrast to the boilerplatecontent referenced and shown above, the content redacted from FIG. 3Avaries from driver license to driver license, and therefore is notsuitable for designating or locating identifying features common to all(or most) driver licenses for a particular state.

Variable content may therefore be understood as any content that is notboilerplate content, and commonly includes text and photographicfeatures of a document. According to preferred embodiments,content-based detection and reconstruction of objects within image dataas disclosed herein is based on boilerplate content, and not based onvariable content.

Although the exemplary embodiment shown in FIG. 3A is a driver license,this is merely illustrative of the type of feature zones that may bedesignated by a user for purposes of locating and leveraging identifyingfeatures as described herein. In other document types, any equivalenttext, especially field designators, may be utilized.

For instance on credit or debit cards a region depicting a name of theissuing entity (e.g. VISA, Bank of America, etc.) may be a suitablefeature zone, or a region depicting a logo corresponding to the issuingentity, a portion of the card background, a portion of the carddepicting a chip (e.g. for a smartcard, an EMV or other equivalentchip), etc. as would be understood by a person having ordinary skill inthe art upon reading the present descriptions.

For checks, suitable feature zones may include field designators such asthe “MEMO” region of the check, Payee designator “PAY TO THE ORDER OF,”boilerplate text such as bank name or address, etc. Similarly, a regionincluding borders of the bounding box designating the numerical paymentamount for the check may be a suitable feature zone, in moreembodiments.

Similarly, for identification documents such as government-issued IDsincluding social security cards, driver licenses, passports, etc.feature zones may include field designators that appear on therespective type of identification document, may include text such as thedocument title (e.g. “United States of America,” “Passport,” “SocialSecurity,” etc.), may include a seal, watermark, logo, hologram, symbol,etc. depicted on the identifying document, or other suitable staticinformation depicted on a same location and in a same manner ondocuments of the same type.

For forms, again field designators are exemplary feature zones suitablefor locating identifying features, as well as lines (particularlyintersecting lines or lines forming a vertex), boxes, etc. as would beunderstood by a person having ordinary skill in the art upon reading thepresent descriptions.

Preferably, the feature zones defined by the user are defined within areference image, i.e. an image representing the object according to apreferred or desired capture angle, zoom level, object orientation, andmost preferably omitting background textures. Advantageously, definingthe feature zones in a reference image significantly reduces the amountof training data necessary to accomplish efficient, accurate, andprecise object detection and three-dimensional reconstruction. Indeed,it is possible to utilize a single training example such as shown inFIG. 3A in various embodiments. Reconstruction shall be discussed infurther detail below.

To determine identifying features within the feature zones, or withinthe image as a whole, a feature vector-based approach is preferablyimplemented. As understood herein, a feature vector is a n-dimensionalvector representing characteristics of a pixel within digital imageand/or video data. The feature vector may include informationrepresentative of the pixel intensity in one or more color channels,pixel brightness, etc. as would be understood by a person havingordinary skill in the art upon reading the present descriptions.

Preferably, identifying features are characterized by a pixel in a smallwindow of pixels (e.g. 8×8, 15×15, or other suitable value which may beconfigured based on image resolution) exhibiting a sharp transition inintensity. The identifying features may be determined based on analyzingthe feature vectors of pixels in the small window, also referred toherein as a “patch.” Frequently, these patches are located in regionsincluding connected components (e.g. characters, lines, etc.) exhibitinga bend or intersection, e.g. as illustrated in FIG. 3B via identifyingfeatures 306 (white dots).

Of course, identifying features and/or feature zones may also bedetermined automatically without departing from the scope of thepresently disclosed inventive concepts, but it should be noted that suchapproaches generally require significantly more training examples thanapproaches in which feature zones are defined manually in a referenceimage. Automatically identifying feature zones may also result in aseries of identifying features 306 as shown in FIG. 3B, in someapproaches.

The aim of automatic feature zone discovery is to find feature pointswithout manually labeling. For instance, in one exemplary embodimentautomatically identifying feature zones may include one or more of thefollowing features and/or operations.

In one approach, the algorithm of selecting feature points involves twopasses. The first pass of the algorithm includes: pair matching,designation of matching points; determining the set of most frequentlyused matching points; and selecting the best image index.

Pair matching may involve assuming a set of cropped images, forinstance, assume a set of ten cropped images denoted by c₁, c₂, c₃, . .. c₁₀, where at least one image is a reference image. From the assumedset, form a set of image pairs preferably including the reference as oneof the images in each image pair. For instance if c₁ is used as thereference image, image pairs may include (c₁, c2), (c₁, c3) . . . (c₁,c₁₀). In addition, for each pair (c₁, c_(k)) (k=2 . . . 10) pairmatching includes finding matching key points between the images, e.g.as described above.

Designating matching points may involve denoting the set of matchingpoints appearing in image c₁ as S_(k), i.e., the set S_(k) includes theset of points in image c₁ that match to their corresponding points inimage c_(k). Designating matching points may also involve denoting theset of matching points in image c_(k) that correspond to the matchingpoints in S_(k) as the set T_(k).

Finding the most frequently used points S_(k) (k=2, 3 . . . 10) may, inturn, include the following. For each point in {S_(k)}(k=2, 3 . . . 10),compute the frequency with which the point is used in {S_(k)}. If thefrequency is above a threshold, for example, 35%, the point is labeledas a “most frequently used” point. In this way, the set of “mostfrequently used” points in image c₁ may be determined, and this set ofpoints is preferably used as the “automatically selected” feature pointsin image c₁. The first pass of the automatic feature identificationalgorithm may also include denoting the selected most commonly usedpoints for image c_(k), as m_(k).

Selecting the best image, in various approaches, may include determiningthe image with the best image index, i.e. the image exhibiting themaximum value of m_(k) (k=1, 2 . . . 10) among images c₁, c₂, . . . c₁₀.

FIG. 3B shows exemplary points 306 automatically selected byimplementing the above algorithm, according to one embodiment.

However, in some approaches the above algorithm may generate featurepoint sets that are more conservative, which means that although theprecision may be high, the recall may be low. Low recall can beproblematic when attempting to match images with a small number ofidentifying features, superimposed against a particularly complexbackground, etc. as would be understood by a person having ordinaryskill in the art upon reading the present disclosures. Accordingly, insome approaches the automatic feature discovery process may include asecond pass aimed at increasing recall of feature point selection.

In a preferred embodiment, the second pass may proceed as follows.Without loss of any generality, suppose that the best image index is 1,that m₁ has the maximum value among different values of m_(k) (k=1, 2 .. . 10), and that this image index represents an undesirably low recallrate. Accordingly, to improve recall, extend the set m₁ by adding moreselected feature points in image c1. The added features may becharacterized by a frequency less than the frequency threshold mentionedabove with regard to the first pass, in some embodiments.

Note that the points in the set m_(k) belongs to image c_(k). For eachm_(k) (k=2 . . . 10), find the corresponding matching points in c₁.Denote as the set of corresponding feature point as v_(k) for each m_(k)where (k=2, 3 . . . 10). The final extended set of selected featurepoints for image c₁ may be defined as the union of m₁, v₂, v₃ . . . andv₁₀. The extended set of selected feature points is shown in FIG. 3C,according to one embodiment. Compared with FIG. 3B, the result shown inFIG. 3C contains more feature points, reflecting the improved recall ofthe second pass.

It should be noted that, in some approaches, automatic feature zonediscovery may be characterized by a systematic bias when operating oncropped images. When observing the layout of text zones or texture zonesin different cropped images of the same object, or objects in the samecategory, there are often variations in layout. There are about 4% to 7%relative changes in locations between different images. The reason forthese variations was not only varying angles or 3D distortions, but alsodue to error inherent to the manufacturing process. In other words, thelocations of particular features often are printed at differentpositions, so that even a scanned image of two different objects of thesame type could exhibit some shift in feature location and/orappearance.

The above problem means the generated models may contain systematicbias. In preferred approaches, it is therefore advantageous to implementan algorithm to compensate for such bias. For instance, the bias may beestimated by the mean value of point shifts in different pair images.For instance, if c₁ is the best selected image. The average value ofpoint shift between each pair image (c₁, c₂), (c₁, c₃) . . . (c₁, c₁₀)is estimated as the bias. Using this approach, it is possible to accountfor bias inherent in the automatic feature zone discovery process asdescribed herein.

Feature vectors may be defined using a suitable algorithm, and in oneembodiment a Binary Robust Independent Elementary Feature (BRIEF) is onesuitable method to define a feature vector or descriptor for a pixel inan image. BRIEF uses grayscale image data as input, but in variousembodiments other color depth input image data, and/or other featurevector defining techniques, may be utilized without departing from thescope of the present descriptions.

In one embodiment, the first step in this algorithm is to remove noisefrom the input image. This may be accomplished using a low-pass filterto remove high frequency noise, in one approach.

The second step is the selection of a set of pixel pairs in the imagepatch around a pixel. For instance, in various approaches pixel pairsmay include immediately adjacent pixels in one or more of four cardinaldirections (up, down, left, and right) and/or diagonally adjacentpixels.

The third step is the comparison of image intensities of each pixelpair. For instance, for a pair of pixels (p, q), if the intensity atpixel p is less than that at pixel q, the comparison result is 1.Otherwise, the result of the comparison is 0. These comparisonoperations are applied to all selected pixel pairs, and a feature vectorfor this image patch is generated by concatenating these 0/1 values in astring.

Assuming a patch comprising 64 pixels, the patch feature vector can havea length of 128, 256, 512, etc. in various approaches and depending onthe nature of the comparison operations. In a preferred embodiment, thefeature vector of the patch has a length of 256, e.g. for a patchcomprising a square 8 pixels long on each side and in which fourcomparisons are performed for each pixel in the patch (left, right,upper and lower neighbor pixels).

A patch descriptor is a representation of a feature vector at a pixel inan image. The shape of a patch around a pixel is usually square orrectangular, but any suitable shape may be employed in various contextsor applications, without departing from the scope of the presentlydisclosed inventive concepts.

In some embodiments, and as noted above the value of each element in afeature vector descriptive of the patch is either 1 or 0, in which casethe descriptor is a binary descriptor. Binary descriptors can berepresented by a string of values, or a “descriptor string.”

As described herein, a descriptor string is analogous to a word innatural language. It can also be called a “visual word.” Similarly, animage is analogous to a document which is characterized by including aparticular set of visual words. These visual words include features thatare helpful for tasks such as image alignment and image recognition. Forinstance, for image alignment, if there are distinctive visual words intwo images, aligning the images based on matching the visual words iseasier than attempting to align the images de novo.

The distance between two descriptor strings can be measured by an editdistance or a Hamming distance, in alternative embodiments. Determiningdistance is a useful indicator of whether two different images, e.g. areference image and a test image, depict similar content at particularpositions. Thus, two images with very small distance between descriptorstrings corresponding to identifying features of the respective imagesare likely to match, especially if the spatial distribution of theproximate identifying features is preserved between the images.

In the original implementation of a BRIEF descriptor algorithm fordefining patch feature vectors, there are no patch orientations, whichmeans that the descriptor is not rotation invariant. However, patchorientations are important to generate patch descriptors which areinvariant to image rotations. Accordingly, in preferred approaches thefeature vector, e.g. BRIEF descriptors, are enhanced with patchorientations which can be estimated using patch momentum. Patch momentummay be analyzed using any suitable technique that would be understood bya person having ordinary skill in the art upon reading the presentdisclosures.

In one embodiment, an “oriented Features from Accelerated Segment Test(FAST) and rotated BRIEF” (ORB) algorithm may be employed to enhancedescriptors with orientation information. After getting the patchorientations, each descriptor is normalized by rotating the image patchwith the estimated rotation angle.

As noted above regarding FIGS. 3A-3C, in preferred approaches the imageincludes one or more identifying features 306, which are characterizedby a sharp transition in pixel intensity within a patch. Accordingly,the position of these identifying features 306 (which may also beconsidered distinctive visual words or key points) is determined.

Key point selection includes finding pixels in an image that havedistinctive visual features. These pixels with distinctive features areat positions where image intensities change rapidly, such as corners,stars, etc. Theoretically speaking, every pixel in an image can beselected as a key point. However, there may be millions of pixels in animage, and using all pixels in image matching is very computationallyintensive, without providing a corresponding improvement to accuracy.Therefore, distinctive pixels, which are characterized by being in apatch exhibiting a rapid change in pixel intensity, are a suitable setof identifying features with which to accurately match images whilemaintaining reasonable computational efficiency. In one embodiment, aFAST (Features from Accelerated Segment Test) algorithm may beimplemented to select key points in image data and/or video data.

In various approaches, image descriptors that are described in theprevious sections are not scale invariant. Therefore, the scale of atraining image and a testing image should be the same in order to findthe best match. For a reference image, a priori knowledge regarding thephysical size of the object and image resolution may be available. Insuch embodiments, it is possible and advantageous to estimate the DPI inthe reference image. Notably, in some approaches using a high resolution(e.g. 1920×1080 or greater, 200 DPI or greater) training image mayproduce too many key points which will slow down image matching process.

In order to optimize the matching time and accuracy, an appropriatereduced DPI level of image/video data is used, in some approaches.Accordingly, for high resolution training images, it is beneficial toscale down to a smaller image resolution, e.g. with a specific DPIlevel. For instance, the reduced DPI level is 180 in one embodimentdetermined to function well in matching images of driver licenses,credit cards, business cards, and other similar documents.

For a test image, the DPI of an object to be detected or matched isgenerally not known. In order to account for this potential variation,it is useful to define a range that the actual image/video dataresolution may reasonably fall within. In one embodiment, this may beaccomplished substantially as follows. The range of resolution valuesmay be quantized with a set of values, in some approaches. For instance,if the resolution range is in a search interval (a, b), where a and bare minimum and maximum DPI values respectively, then the interval (a,b) are divided into a set of sub intervals. The test image is scaleddown to a set of images with different, but all reduced, resolutions,and each re-scaled image is matched to the training image. The bestmatch found indicates the appropriate downscaling level.

The detail of a matching algorithm, according to one embodiment, is asfollows. For each resolution in the search interval: a test image isscaled down to the resolution used in the reference image. A brute-forcematching approach may be employed to identify the matching pointsbetween the reference image and test image. The key points in thereference image are matched against some, or preferably all, key pointsidentified in the testing image. First, the best match for each keypoint both in the reference image and test image is identified bycomparing the distance ratio of the two best candidate matches. When thedistance ratio is larger than a predetermined threshold, the match isidentified as an outlier.

After distance ratio testing, in some embodiments a symmetrical matchingtest may be applied to further identify other potential remainingoutliers. In the symmetrical matching test, if the match between keypoints in the reference image and test image is unique (i.e. the keypoints in the reference and test image match one another, but do notmatch any other key points in the corresponding image), then the keypoints will be kept. If a match between corresponding key point(s) inthe reference image and test image is not unique, those key points willbe removed.

After performing brute-forced matching, there are still potentialoutliers in the remaining matches. Accordingly, an outlieridentification algorithm such as a Random Sample Consensus (RANSAC)algorithm is applied to further remove outliers. The details of RANSACalgorithm are summarized below. In one embodiment implementing theRANSAC algorithm, the best match is found, and the number of matchingkey points is recorded.

RANSAC is a learning technique to estimate parameters of a model byrandom sampling of observed data. For plane image matching tasks, suchas documents, the model is a homograph transformation of a 3 by 3matrix.

In one embodiment, the RANSAC algorithm to estimate the homographtransformation is as follows. First, randomly select four key points ina testing image, and randomly select four key points in a referenceimage. Second, estimate a homograph transform with the above four keypoint pairs using a four-point algorithm, e.g. as described belowregarding image reconstruction. Third, apply the homographtransformation to all key points in the reference and testing images.The inlier key points are identified if they match the model well,otherwise the key points will be identified as outliers. In variousembodiments, more than four points may be selected for this purpose, butpreferably four points are utilized to minimize computational overheadwhile enabling efficient, effective matching.

The foregoing three-step process is repeated in an iterative fashion tore-sample the key points and estimate a new homograph transform. In oneembodiment, the number of iterations performed may be in a range fromabout 10²-10³ iterations. After the iterative identification of keypoints is complete, the largest inlier set is retained, and an affine orhomograph transform is re-estimated based on the retained inlier set.

After removing outliers, the matching process selects the referenceimage with the maximum number of matching points as the best match, andan affine or homograph transform is estimated with the best match toreconstruct the image and/or video data in a three-dimensionalcoordinate system. Image reconstruction mechanics are discussed infurther detail below.

Exemplary mappings of key points between a reference image 400 and testimage 410 are shown, according to two embodiments, in FIGS. 4A-4B, withmapping lines 402 indicating the correspondence of key points betweenthe two images. FIG. 4C depicts a similar reference/test image pair,showing a credit or debit card and exemplary corresponding key pointstherein, according to another embodiment.

Advantageously, by identifying internal key points and mapping keypoints located in a test image 410 to corresponding key points in areference image 400, the presently disclosed inventive concepts candetect objects depicted in image and/or video data even when the edgesof the object are obscured or missing from the image, or when a complexbackground is present in the image or video data.

Once an appropriate transform is estimated, the presently disclosedinventive concepts advantageously allow the estimation of objectedge/border locations based on the transform. In brief, based on theedge locations determined from the reference image data, it is possibleto estimate the locations of corresponding edges/borders in the testimage via the transform, which defines the point-to-point correspondencebetween the object as oriented in the test image and a correspondingreference image orientation within the same coordinate system. Accordingto the embodiment shown in FIGS. 4A and 4B, estimating the edgelocations involves evaluating the transform of the document plane shownin test image 410 to the document plane shown in the reference image 400(or vice versa), and extrapolating edge positions based on thetransform.

FIG. 4C shows a similar mapping of key points between a reference image400 and test image 410 of a credit card. In the particular case ofcredit cards, and especially credit cards including an IC chip, it ispossible to identify key points within the region of the card includingthe IC chip, and estimate transform(s) and/or border locations usingthese regions as the sole source of key points, in various embodiments.Accordingly, the presently disclosed inventive concepts are broadlyapplicable to various different types of objects and identifyingfeatures, constrained only by the ability to obtain and identifyappropriate identifying features in a suitable reference image or set ofreference images. Those having ordinary skill in the art will appreciatethe scope to which these inventive concepts may be applied upon readingthe instant disclosures.

Based on the transform, and the projected object edges, the presentlydisclosed inventive concepts may include transforming and cropping thetest image to form a cropped, reconstructed image based on the testimage, the cropped, reconstructed image depicting the object accordingto a same perspective and cropping of the object as represented in thereference image perspective.

In addition, preferred embodiments may include functionality configuredto refine the projected location of object edges. For example,considering the results depicted in FIGS. 4A-4C and 5, a skilled artisanwill understand that the projected edges achieved in these exemplaryembodiments are not as accurate as may be desired.

As shown in FIG. 5, an object 500 such as a credit card or otherdocument is depicted in a test image, and edge locations 502 areprojected based on the foregoing content-based approach. However, theprojected edge locations 502 do not accurately correspond to the actualedges of the object 500. Accordingly, it may be advantageous, in someapproaches, rather than cropping directly according to the projectededge locations 502, to crop in a manner so as to leave a predeterminedamount of background texture depicted in the cropped image, andsubsequently perform conventional edge detection. Conventional edgedetection shall be understood to include any technique for detectingedges based on detecting transitions between an image background 504 andimage foreground (e.g. object 500) as shown in FIG. 5. For example, inpreferred approaches conventional edge detection may include anytechnique or functionality as described in U.S. Pat. No. 8,855,375 toMacciola, et al.

The predetermined amount may be represented by a threshold ∂, which maybe a predefined number of pixels, a percentage of an expected aspectratio, etc. in various embodiments. In some approaches, the amount maybe different for each dimension of the image and/or object, e.g. forflat objects a predetermined height threshold ∂_(H) and/or predeterminedwidth threshold ∂_(W) may be used. ∂_(H) and ∂_(W) may be determinedexperimentally, and need not be equal in various embodiments. Forinstance, ∂_(H) and ∂_(W) may independently be absolute thresholds orrelative thresholds, and may be characterized by different values.

In this way, one obtains an image where the document is prominent in theview and the edges reside within some known margin. Now it is possibleto employ normal or specialized edge detection techniques, which mayinclude searching for the edge only within the margin. In “normal”techniques, the threshold for detection can be less stringent thantypically employed when searching for edges using only a conventionalapproach, without content-based detection augmentation. For instance, in“normal” techniques the contrast difference required to identify an edgemay be less than the difference required without content-based detectionaugmentation. In “specialized” techniques, one could allow for increasedtolerance regarding existence of gaps in the edge than would normally beprudent when searching an entire image (e.g. as would be present in FIG.4A).

In various approaches, a further validation may be performed on theimage and/or video data by classifying the cropped, reconstructed image.Classification may be performed using any technique suitable in the art,preferably a classification technique as described in U.S. patentapplication Ser. No. 13/802,226 (filed Mar. 13, 2013). If theclassification result returns the appropriate object type, then theimage matching and transform operations are likely to have beencorrectly achieved, whereas if a different object type is returned fromclassification, then the transform and/or cropping result are likelyerroneous. Accordingly, the presently disclosed inventive concepts mayleverage classification as a confidence measure to evaluate the imagematching and reconstruction techniques discussed herein.

As described herein, according to one embodiment a method 700 fordetecting objects depicted in digital images based on internal featuresof the object includes operations as depicted in FIG. 7. As will beunderstood by a person having ordinary skill in the art upon reading thepresent descriptions, the method 700 may be performed in any suitableenvironment, including those depicted in FIGS. 1-2 and may operate oninputs and/or produce outputs as depicted in FIGS. 3A-5, in variousapproaches.

As shown in FIG. 7, method 700 includes operation 702, in which aplurality of identifying features of the object are detected. Notably,the identifying features are located internally with respect to theobject, such that each identifying feature is, corresponds to, orrepresents a part of the object other than object edges, boundariesbetween the object and image background, or other equivalent transitionbetween the object and image background. In this manner, and accordingto various embodiments the presently disclosed inventive content-basedobject recognition techniques are based exclusively on the content ofthe object, and/or are performed exclusive of traditional edgedetection, border detection, or other similar conventional recognitiontechniques known in the art.

The method 700 also includes operation 704, where a location of one ormore edges of the object are projected, the projection being based atleast in part on the plurality of identifying features.

Of course, the method 700 may include any number of additional and/oralternative features as described herein in any suitable combination,permutation, selection thereof as would be appreciated by a skilledartisan as suitable for performing content-based object detection, uponreading the instant disclosures.

For instance, in one embodiment, method 700 may additionally oralternatively include detecting the plurality of identifying featuresbased on analyzing a plurality of feature vectors each corresponding topixels within a patch of the digital image. The analysis may beperformed in order to determine whether the patch includes a sharptransition in intensity, in preferred approaches. The analysis mayoptionally involve determining a position of some or all of theplurality of identifying features, or position determination may beperformed separately from feature vector analysis, in variousembodiments.

Optionally, in one embodiment detecting the plurality of identifyingfeatures involves automatic feature zone discovery. The automaticfeature zone discovery may be a multi-pass procedure.

Method 700 may also include identifying a plurality of distinctivepixels within the plurality of identifying features of the object.Distinctive pixels are preferably characterized by having or embodyingdistinct visual features of the object.

In a preferred approach, method 700 also includes matching the digitalimage depicting the object to one of a plurality of reference imageseach depicting a known object type. The reference images are morepreferably images used to train the recognition/detection engine toidentify specific identifying features that are particularly suitablefor detecting and/or reconstructing objects of the known object type invarious types of images and/or imaging circumstances (e.g. differentangles, distances, resolutions, lighting conditions, color depths, etc.in various embodiments). Accordingly, the matching procedure may involvedetermining whether the object includes distinctive pixels thatcorrespond to distinctive pixels present in one or more of the pluralityof reference images.

The method 700 may also include designating as an outlier a candidatematch between a distinctive pixel in the digital image and one or morecandidate corresponding distinctive pixels present in one of theplurality of reference images. The outlier is preferably designated inresponse to determining a distance ratio is greater than a predetermineddistance ratio threshold. Moreover, the distance ratio may be a ratiodescribing: a first distance between the distinctive pixel in thedigital image and a first of the one or more candidate correspondingdistinctive pixels; and a second distance between the distinctive pixelin the digital image and a second of the one or more candidatecorresponding distinctive pixels.

In more embodiments, method 700 includes designating as an outlier acandidate match between a distinctive pixel in the digital image and acandidate corresponding distinctive pixel present in one of theplurality of reference images in response to determining the candidatematch is not unique. Uniqueness may be determined according to asymmetrical matching test, in preferred approaches and as described ingreater detail hereinabove.

Notably, employing reconstruction as set forth herein, particularly withrespect to method 700, carries the advantage of being able to detect andrecognize objects in images where at least one edge of the object iseither obscured or missing from the digital image. Thus, the presentlydisclosed inventive concepts represent an improvement to imageprocessing machines and the image processing field since conventionalimage detection and image processing/correction techniques are based ondetecting the edges of objects and making appropriate corrections basedon characteristics of the object and/or object edges (e.g. locationwithin image, dimensions such as particularly aspect ratio, curvature,length, etc.). In image data where edges are missing, obscured, orotherwise not represented at least in part, such conventional techniqueslack the requisite input information to perform the intended imageprocessing/correction.

In some approaches, the method 700 may include cropping the digitalimage based at least in part on the projected location of the one ormore edges of the object. The cropped digital image preferably depicts aportion of a background of the digital image surrounding the object; andin such approaches method 700 may include detecting one or moretransitions between the background and the object within the croppeddigital image.

The method 700 may optionally involve classifying the object depictedwithin the cropped digital image. As described in further detailelsewhere herein, classification may operate as a type of orthogonalvalidation procedure or confidence measure for determining whether imagerecognition and/or reconstruction was performed correctly byimplementing the techniques described herein. In brief, if areconstructed image of an object is classified and results in adetermination that the object depicted in the reconstructed image is asame type of object represented in/by the reference image used toreconstruct the object, then it is likely the reconstruction wasperformed correctly, or at least optimally under the circumstances ofthe image data.

With continuing reference to classification, method 700 in oneembodiment may include: attempting to detect the object within thedigital image using a plurality of predetermined object detection modelseach corresponding to a known object type; and determining aclassification of the object based on a result of attempting to detectthe object within the digital image using the plurality of predeterminedobject detection models. The classification of the object is the knownobject type corresponding to one of the object detection models forwhich the attempt to detect the object within the digital image wassuccessful.

The method 700, in additional aspects, may include: generating aplurality of scaled images based on the digital image, each scaled imagebeing characterized by a different resolution; extracting one or morefeature vectors from each scaled image; and matching one or more of thescaled images to one of a plurality of reference images. Each referenceimage depicts a known object type and being characterized by a knownresolution.

Of course, in various embodiments and as described in greater detailbelow, the techniques and features of method 700 may be combined andused to advantage in any permutation with the various imagereconstruction techniques and features such as presented with respect tomethod 800.

Content-Based Image Reconstruction

Reconstructing image and/or video data as described herein essentiallyincludes transforming the representation of the detected object asdepicted in the captured image and/or video data into a representationof the object as it would appear if viewed from an angle normal to aparticular surface of the object. In the case of documents, or otherflat objects, this includes reconstructing the object representation toreflect a face of the flat object as viewed from an angle normal to thatface. For such flat objects, if the object is characterized by a knowngeometry (e.g. a particular polygon, circle, ellipsoid, etc.) then apriori knowledge regarding the geometric characteristics of the knowngeometry may be leveraged to facilitate reconstruction

For other objects having three-dimensional geometries, and/or flatobjects having non-standard geometries, reconstruction preferablyincludes transforming the object as represented in captured image and/orvideo data to represent a same or similar object type as represented inone or more reference images captured from a particular angle withrespect to the object. Of course, reference images may also be employedto facilitate reconstruction of flat objects in various embodiments andwithout departing from the scope of the presently disclosed inventiveconcepts.

Accordingly, in preferred approaches the reconstructed representationsubstantially represents the actual dimensions, aspect ratio, etc. ofthe object captured in the digital image when viewed from a particularperspective (e.g. at an angle normal to the object, such as would be thecapture angle if scanning a document in a traditional flatbed scanner,multifunction device, etc. as would be understood by one having ordinaryskill in the art upon reading the present descriptions).

Various capture angles, and the associated projective effects aredemonstrated schematically in FIGS. 6A-6D.

In some approaches, the reconstruction may include applying an algorithmsuch as a four-point algorithm to the image data.

In one embodiment, perspective correction may include constructing a 3Dtransformation based at least in part on the spatial distribution offeatures represented in the image and/or video data.

A planar homography/projective transform is a non-singular linearrelation between two planes. In this case, the homography transformdefines a linear mapping of four randomly selected pixels/positionsbetween the captured image and the reference image.

The calculation of the camera parameters may utilize an estimation ofthe homography transform H, such as shown in Equation (1), in someapproaches.

$\begin{matrix}{{\lambda \begin{pmatrix}x \\y \\1\end{pmatrix}} = {\underset{\underset{{homography}\mspace{14mu} H}{}}{\begin{pmatrix}h_{11} & h_{12} & h_{13} \\h_{21} & h_{22} & h_{23} \\h_{31} & h_{32} & h_{33}\end{pmatrix}}{\begin{pmatrix}X \\Y \\1\end{pmatrix}.}}} & (1)\end{matrix}$

As depicted above in Equation (1):

-   -   λ is the focal depth of position (X, Y, Z) in the “reference” or        “real-world” coordinate system, (e.g. a coordinate system        derived from a reference image,). Put another way, λ may be        considered the linear distance between a point (X,Y,Z) in the        reference coordinate system and the capture device;    -   (x, y, z) are the coordinates of a given pixel position in the        captured image; and    -   H is a (3×3) matrix having elements h_(ij), where, i and j        define the corresponding row and column index, respectively.

In one approach, the (x, y) coordinates and (X, Y) coordinates depictedin Equation 1 correspond to coordinates of respective points in thecaptured image plane and the reference image. In some approaches, the Zcoordinate may be set to 0, corresponding to an assumption that theobject depicted in each lies along a single (e.g. X-Y) plane with zerothickness. In one embodiment, it is possible to omit the z value inEquation 1 from the above calculations because it does not necessarilyplay any role in determining the homography matrix.

Thus, the homography H can be estimated by detecting fourpoint-correspondences p_(i)↔P_(i)′ with p_(i)=(x_(i), y_(i), 1)^(T)being the position of a randomly selected feature in the captured imageplane; and P_(i)′=(X_(i), Y_(i), 1)^(T) being the coordinates of thecorresponding position in the reference image, where i is point indexvalue with range from 1 to n in the following discussion. Using thepreviously introduced notation, Equation (1) may be written as shown inEquation (2) below.

Δp _(i) =HP _(i)′,  (2)

In order to eliminate a scaling factor, in one embodiment it is possibleto calculate the cross product of each term of Equation (2), as shown inEquation (3):

p _(i)×(λp _(i))=p _(i)×(HP _(i)′),  (3)

Since p_(i)×p_(i)=0₃, Equation (3) may be written as shown below inEquation (4).

p _(i) ×HP _(i)′=0₃,  (4)

Thus, the matrix product HP_(i)′ may be expressed as in Equation (5).

$\begin{matrix}{{{HP}_{i}^{\prime} = \begin{bmatrix}{h^{1\; T}P_{i}^{\prime}} \\{h^{2\; T}P_{i}^{\prime}} \\{h^{3\; T}P_{i}^{\prime}}\end{bmatrix}},} & (5)\end{matrix}$

According to Equation 5, h^(mT) is the transpose of the m^(th) row of H(e.g. h^(1T) is the transpose of the first row of H, h^(2T) is thetranspose of the second row of H, etc.). Accordingly, it is possible torework Equation (4) as:

$\begin{matrix}{{p_{i} \times {HP}_{i}^{\prime}} = {{\begin{pmatrix}x_{i} \\y_{i} \\1\end{pmatrix} \times \begin{bmatrix}{h^{1\; T}P_{i}^{\prime}} \\{h^{2\; T}P_{i}^{\prime}} \\{h^{3\; T}P_{i}^{\prime}}\end{bmatrix}} = {\begin{bmatrix}{{y_{i}h^{3\; T}P_{i}^{\prime}} - {h^{2\; T}P_{i}^{\prime}}} \\{{h^{1\; T}P_{i}^{\prime}} - {x_{i}h^{3\; T}P_{i}^{\prime}}} \\{{x_{i}h^{2\; T}P_{i}^{\prime}} - {y_{i}h^{1\; T}P_{i}^{\prime}}}\end{bmatrix} = {0_{3}.}}}} & (6)\end{matrix}$

Notably, Equation (6) is linear in h^(mT) andh^(mT)P_(i)′=P_(i)′^(T)h^(m). Thus, Equation (6) may be reformulated asshown below in Equation (7):

$\begin{matrix}{{\begin{bmatrix}0_{3}^{T} & {- P_{i}^{\prime}} & {y_{i}P_{i}^{\prime \; T}} \\P_{i}^{\prime \; T} & 0_{3}^{T} & {{- x_{i}}P_{i}^{\prime \; T}} \\{{- y_{i}}P_{i}^{\prime \; T}} & {x_{i}P_{i}^{\prime \; T}} & 0_{3}^{T}\end{bmatrix}\begin{bmatrix}h^{1} \\h^{2} \\h^{3}\end{bmatrix}} = {0_{3}.}} & (7)\end{matrix}$

Note that the rows of the matrix shown in Equation (7) are not linearlyindependent. For example, in one embodiment the third row is the sum of−x_(i) times the first row and −y_(i) times the second row. Thus, foreach point-correspondence, Equation (7) provides two linearlyindependent equations. The two first rows are preferably used forsolving H.

Because the homography transform is written using homogeneouscoordinates, in one embodiment the homography H may be defined using 8parameters plus a homogeneous scaling factor (which may be viewed as afree 9^(th) parameter). In such embodiments, at least 4point-correspondences providing 8 equations may be used to compute thehomography. In practice, and according to one exemplary embodiment, alarger number of correspondences is preferably employed so that anover-determined linear system is obtained, resulting in a more robustresult (e.g. lower error in relative pixel-position). By rewriting H ina vector form as h=[h₁₁,h₁₂,h₁₃,h₂₁,h₂₂,h₂₃,h₃₁,h₃₂,h₃₃]^(T), n pairs ofpoint-correspondences enable the construction of a 2n×9 linear system,which is expressed by Equation (8)

$\begin{matrix}{{\underset{\underset{C}{}}{\begin{pmatrix}0 & 0 & 0 & {- X_{1}} & {- Y_{1}} & {- 1} & {y_{1}X_{1}} & {y_{1}X_{1}} & y_{1} \\X_{1} & Y_{1} & 1 & 0 & 0 & 0 & {{- x_{1}}X_{1}} & {{- x_{1}}Y_{1}} & {- x_{1}} \\0 & 0 & 0 & {- X_{2}} & {- Y_{2}} & {- 1} & {y_{2}X_{2}} & {y_{2}X_{2}} & y_{2} \\X_{2} & Y_{2} & 1 & 0 & 0 & 0 & {{- x_{2}}X_{2}} & {{- x_{2}}Y_{2}} & {- x_{2}} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\0 & 0 & 0 & {- X_{n}} & {- Y_{n}} & {- 1} & {y_{n}X_{n}} & {y_{n}X_{n}} & y_{n} \\X_{n} & Y_{n} & 1 & 0 & 0 & 0 & {{- x_{n}}X_{n}} & {{- x_{n}}Y_{n}} & {- x_{n}}\end{pmatrix}}\begin{pmatrix}h_{11} \\h_{12} \\h_{13} \\h_{21} \\h_{22} \\h_{23} \\h_{31} \\h_{32} \\h_{33}\end{pmatrix}} = {0_{3}.}} & (8)\end{matrix}$

As shown in Equation 8, the first two rows correspond to the firstfeature point, as indicated by the subscript value of coordinates X, Y,x, y—in this case the subscript value is 1. The second two rowscorrespond to the second feature point, as indicated by the subscriptvalue 2, the last two rows correspond to the n^(th) feature point. For afour-point algorithm, n is 4, and the feature points are the fourrandomly selected features identified within the captured image andcorresponding point of the reference image.

In one approach, solving this linear system involves the calculation ofa Singular Value Decomposition (SVD). Such an SVD corresponds toreworking the matrix to the form of the matrix product C=UDV^(T), wherethe solution h corresponds to the eigenvector of the smallest eigenvalueof matrix C, which in one embodiment may be located at the last columnof the matrix V when the eigenvalues are sorted in descendant order.

It is worth noting that the matrix C is different from the typicalmatrix utilized in an eight-point algorithm to estimate the essentialmatrix when two or more cameras are used, such as conventionallyperformed for stereoscopic machine vision. More specifically, while theelements conventionally used in eight-point algorithm consist of featurepoints projected on two camera planes, the elements in the presentlydescribed matrix C consist of feature points projected on only a singlecamera plane and the corresponding feature points on 3D objects.

In one embodiment, to avoid numerical instabilities, the coordinates ofpoint-correspondences may preferably be normalized. This may beaccomplished, for example, using a technique known as the normalizedDirect Linear Transformation (DLT) algorithm. For example, in oneembodiment, after the homography matrix is estimated, Equation 1 may beused to compute each pixel position (x, y) for a given value of (X, Y).In practical applications the challenge involves computing (X, Y) whenthe values of (x, y) are given or known a priori. As shown in Equation1, and in preferred embodiments, (x, y) and (X, Y) are symmetrical (i.e.when the values of (x, y) and (X, Y) are switched, the validity ofEquation 1 holds true). In this case, the “inverse” homography matrixmay be estimated, and this “inverse” homography matrix may be used toreconstruct 3D (i.e. “reference” or “real-world”) coordinates of anobject given the corresponding 2D coordinates of the object as depictedin the captured image, e.g. in the camera view.

Based on the foregoing, it is possible to implement the presentlydescribed four-point algorithm (as well as any equivalent variationand/or modification thereof that would be appreciated by a skilledartisan upon reading these descriptions) which may be utilized invarious embodiments to efficiently and effectively reconstruct digitalimages characterized by at least some perspective distortion intocorrected digital images exempting any such perspective distortion,where the corrected image is characterized by a pixel location error ofabout 5 pixels or less.

Various embodiments may additionally and/or alternatively includeutilizing the foregoing data, calculations, results, and/or concepts toderive further useful information regarding the captured image, object,etc. For example, in various embodiments it is possible to determine thedistance between the captured object and the capture device, the pitchand/or roll angle of the capture device, etc. as would be understood byone having ordinary skill in the art upon reading the presentdescriptions.

After (X, Y) values are estimated, the expression in Equation 1 may bedescribed as follows:

λ=h ₃₁ X+h ₃₂ Y+h ₃₃  (9)

Accordingly, in one embodiment the focal depth, also known as thedistance between each point (X, Y, Z) in the 3D (i.e. “reference” or“real world”) coordinate system and the capture device, may be computedusing Equation 9 above.

Determining a Rotation Matrix of the Object.

After estimating the position of the 3D object (X, Y) and λ for eachpixel in the captured image. Note that (X, Y) are the coordinates in theworld coordinate system, while 2 is the distance to the point (X, Y) inthe camera coordinate system. If the 3D object is assumed to be a rigidbody, it is appropriate to use the algorithm disclosed herein toestimate the rotation matrix from the world coordinate system to thecamera coordinate system. The following equation holds for rotation andtranslation of the point (X, Y, 0):

$\begin{matrix}{\begin{pmatrix}X_{c} \\Y_{c} \\Z_{c}\end{pmatrix} = {{R\begin{pmatrix}X \\Y \\0\end{pmatrix}} + t}} & (10)\end{matrix}$

where (Xc, Yc, Zc) are the coordinates relative to camera coordinatesystem, which are derived by rotating a point (X, Y, Z) in the worldcoordinate system with rotation matrix R, and a translation vector of t,where t is a constant independent of (X, Y). Note that the value of Zcis the same as the value of λ, as previously estimated using equation 9.

Considering the relationships of homography matrix H and intrinsiccamera parameter matrix A and r1, r2, where r1, r2 are the first andsecond column vectors respectively, reveals the following relationship:

H=σA(r ₁ ,r ₂ ,t)  (11)

where σ is a constant and A is the intrinsic camera parameter matrix,defined as:

$\begin{matrix}{A = \begin{pmatrix}a & c & d \\\; & b & e \\\; & \; & 1\end{pmatrix}} & (12)\end{matrix}$

where a and b are scaling factors which comprise of the camera focallength information, a=f/dx, and b=f/dy, where f is the focal length,while dx, dy are scaling factors of the image; c is the skew parameterabout two image axes, and (d, e) are the coordinates of thecorresponding principal point.

After estimation of homography matrix H, the matrix A can be estimatedas follows:

$\begin{matrix}{{a = \sqrt{w/B_{11}}};} & (12.1) \\{{b = \sqrt{{wB}_{11}\left( {{B_{11}B_{22}} - B_{12}^{2}} \right)}};} & (12.2) \\{{c = {{- B_{12}}a^{2}{b/w}}};{d = {\frac{{vv}_{0}}{b} - {B_{13}{a^{2}/w}}}};} & (12.3) \\{{v = {{- B_{12}}a^{2}{b/w}}};} & (12.4) \\{{e = {\left( {{B_{12}B_{13}} - {B_{11}B_{23}}} \right)/\left( {{B_{11}B_{22}} - B_{12}^{2}} \right)}};} & (12.5) \\{w = {B_{33} - {\left( {B_{13}^{2} + {e\left( {{B_{12}B_{13}} - {B_{11}B_{23}}} \right)}} \right)/{B_{11}.}}}} & (12.6)\end{matrix}$

In the above relationships, the unknown parameters are B_(ij). Thesevalues are estimated by the following equations:

$\begin{matrix}{{{\begin{pmatrix}v_{12}^{t} \\\left( {v_{11} - v_{22}} \right)^{t}\end{pmatrix}G} = 0},} & (12.7)\end{matrix}$

where G is the solution of the above equation, alternatively expressedas:

G=(B ₁₁ ,B ₁₂ ,B ₂₂ ,B ₁₃ ,B ₂₃ ,B ₃₃)^(t),  (12.8)

where v _(ij)=(h _(i1) h _(j1) ,h _(i1) h _(j2) +h _(i2) h _(j1) ,h_(i2) h _(j2) ,h _(i3) h _(j1) +h _(i1) h _(j3) ,h _(i3) h _(j2) +h_(i2) h _(j3) ,h _(i3) h _(j3))^(t)  (12.9)

Note that in a conventional four-points algorithm, since it is possibleto accurately estimate scaling factors a, b, the skew factor c isassumed to be zero, which means that one may ignore camera's skewdistortion. It is further useful, in one embodiment, to assume that dand e have zero values (d=0, e=0).

From equation (11), B=(r1 r2 t), where σ⁻¹ A⁻¹H=B. Utilizing thisrelationship enables a new approach to estimate r1, r2 from the equationC=(r1 r2 0) where the first and second column vectors of C are the firstand second column vectors of B, and the third column vector of C is 0.

First, decompose matrix C with SVD (Singular Value Decomposition)method, C=UΣV^(t), where U is 3 by 3 orthogonal matrix, where V is 3 by3 orthogonal matrix. Then r1 and r2 are estimated by the followingequation:

$\begin{matrix}{\begin{pmatrix}r_{1} & r_{2} & 0\end{pmatrix} = {U\begin{pmatrix}W \\0\end{pmatrix}}} & (13)\end{matrix}$

where W is a 2 by 3 matrix whose first and second row vectors are thefirst and second row vectors of V^(t) respectively. In the abovecomputation, assume σ is 1. This scaling factor does not influence thevalue of U and W and therefore does not influence the estimation of r1and r2. After r1, r2 are estimated (e.g. using Equation 13), it isuseful to leverage the fact that R is a rotation matrix to estimate r3,which is the cross product of r1 and r2 with a sign to be determined(either 1 or −1). There are two possible solutions of R. In one exampleusing a right-hand coordinate system, the r3 value is the cross-productvalue of r1 and r2.

Determining Yaw, Pitch, and Roll from a Rotation Matrix.

The yaw, pitch and roll (denoted by the α, β and γ respectively) arealso known as Euler's angles, which are defined as the rotation anglesaround z, y, and x axes respectively, in one embodiment. According tothis approach, the rotation matrix R in Equation 10 can be denoted as:

$\begin{matrix}{R = \begin{pmatrix}r_{11} & r_{12} & r_{13} \\r_{21} & r_{22} & r_{23} \\r_{31} & r_{32} & r_{33}\end{pmatrix}} & (14)\end{matrix}$

where each r is an element of the matrix R.

It is often convenient to determine the α, β and γ parameters directlyfrom a given rotation matrix R. The roll, in one embodiment, may beestimated by the following equation (e.g. when r₃₃ is not equal tozero):

γ=a tan 2(r ₃₂ ,r ₃₃)  (15)

Similarly, in another approach the pitch may be estimated by thefollowing equation:

β=a tan 2(−r ₃₁,√{square root over (r ₁₁ ² +r ₂₁ ²)})  (16)

In still more approaches, the yaw may be estimated by the followingequation (e.g. when r₁₁ is nonzero)

α=a tan 2(r ₂₁ ,r ₁₁)  (17)

Notably, in some approaches when r₁₁, r₃₃ or √{square root over (r₁₁²+r₂₁ ²)} are near in value to zero (e.g. 0<r₁₁<ε, 0<r₃₃<ε, or0<√{square root over (r₁₁ ²+r₂₁ ²)}<ε, where the value E is set to areasonable value for considering the numerical stability, such as0<ε≤0.01, in one embodiment, and c=0.0001 in a particularly preferredembodiment. In general, the value of c may be determined in whole or inpart based on limited computer word length, etc. as would be understoodby one having ordinary skill in the art upon reading the presentdescriptions), this corresponds to the degenerate of rotation matrix R,special formulae are used to estimate the values of yaw, pitch and roll.

Estimating Distance Between Object and Capture Device

In still more embodiments, it is possible to estimate the distancebetween an object and a capture device even without the knowledge of theobject size, using information such as a camera's intrinsic parameters(e.g. focal length, scale factors of (u, v) in image plane).

The requirements of this algorithm, in one approach, may be summarizedas follows: 1) The camera's focal length for the captured image can beprovided and accessed by an API call of the device (for instance, anandroid device provides an API call to get focal length information forthe captured image); 2) The scale factors of dx and dy are estimated bythe algorithm in the equations 12.1 and 12.2.

This enables estimation of the scale factors dx, dy for a particulartype of device, and does not require estimating scale factors for eachdevice individually. For instance, in one exemplary embodiment utilizingan Apple iPhone® 4 smartphone, it is possible, using the algorithmpresented above, to estimate the scale factors using an object with aknown size. The two scaling factors may thereafter be assumed to beidentical for the same device type.

The algorithm to estimate object distance to camera, according to oneillustrative approach, is described as follows: normalize (u, v), (X, Y)in the equation below

$\begin{matrix}{{\lambda \begin{pmatrix}u \\v \\1\end{pmatrix}} = {H\begin{pmatrix}X \\Y \\1\end{pmatrix}}} & (18)\end{matrix}$

Note that Equation 18 is equivalent to Equation 1, except (u, v) inEquation 18 replaces the (x, y) term in Equation 1.

Suppose that ũ=u/L_(u), {tilde over (v)}=v/L_(v); {tilde over(x)}=X/L_(X); {tilde over (y)}=Y/L_(Y); where L_(u), L_(v) are imagesize in coordinates u and v and L_(X), L_(Y) are the object size to bedetermined.

Then Equation 18 may be expressed as:

$\begin{matrix}{{{\lambda \begin{pmatrix}\overset{\sim}{u} \\\overset{\sim}{v} \\1\end{pmatrix}} = {\overset{\sim}{H}\begin{pmatrix}\overset{\sim}{x} \\\overset{\sim}{y} \\1\end{pmatrix}}},{where}} & (19) \\{\overset{\sim}{H} = {\begin{pmatrix}{1/L_{u}} & \; & \; \\\; & {1/L_{v}} & \; \\\; & \; & 1\end{pmatrix}{H\begin{pmatrix}L_{x} & \; & \; \\\; & L_{y} & \; \\\; & \; & 1\end{pmatrix}}}} & (20)\end{matrix}$

Normalized homography matrix {tilde over (H)} can be estimated byequation (20). Note that from equation 11, the following may bedetermined:

H=σA(r ₁ r ₂ t)  (21)

and the intrinsic parameter matrix of the camera is assumed with thefollowing simple form:

$\begin{matrix}{A = \begin{pmatrix}{f/{dx}} & c & d \\\; & {f/{dy}} & e \\\; & \; & 1\end{pmatrix}} & (22)\end{matrix}$

where f is the camera focal length, dx, dy are scaling factors of thecamera, which are estimated.

From equations (19), (20) and (21), thus:

$\begin{matrix}{{{\sigma \; {A\begin{pmatrix}r_{1} & r_{2} & t\end{pmatrix}}\begin{pmatrix}L_{x} & \; & \; \\\; & L_{y} & \; \\\; & \; & 1\end{pmatrix}} = \overset{\sim}{\overset{\sim}{H}}}{{{where}\mspace{14mu} \overset{\sim}{\overset{\sim}{H}}} = {\begin{pmatrix}L_{u} & \; & \; \\\; & L_{v} & \; \\\; & \; & 1\end{pmatrix}\overset{\sim}{H}}}} & (23)\end{matrix}$

Because A is known, from equation (23) the following may be determined:

$\begin{matrix}{{\sigma \; \begin{pmatrix}r_{1} & r_{2} & t\end{pmatrix}\begin{pmatrix}L_{x} & \; & \; \\\; & L_{y} & \; \\\; & \; & 1\end{pmatrix}} = {A^{- 1}\overset{\sim}{\overset{\sim}{H}}}} & (24)\end{matrix}$

Denote K=A⁻¹{tilde over ({tilde over (H)})}, K=(k₁, k₂, k₃), fromequation (24) the following may be determined:

σr ₁ L _(x) =k ₁  (25)

σr ₂ L _(y) =k ₂  (26)

σ_(t) =k ₃  (27)

because t in equation (27) is the translation vector of the objectrelative to camera. The L2 norm (Euclidean norm) of t is as follows:

∥t∥=∥k ₃∥/σ  (28)

is the distance of left-top corner of the object to the camera.

Because ∥r₁∥=∥r₂∥=1, from equation (8) and (9), the following may bedetermined

L _(x) =∥k ₁∥/σ  (29)

L _(y) =∥k ₂∥/σ  (30)

Equations (29) and (30) may be used to estimate the document size alongX and Y coordinates. The scaling factor may remain unknown, using thisapproach.

Note that the algorithm to estimate rotation matrix described above doesnot need the scaling factor σ. Rather, in some approaches it is suitableto assume σ=1. In such cases, it is possible to estimate roll, pitch,and yaw with the algorithm presented above. Equations (29) and (30) mayalso be used to estimate the aspect ratio of the object as:

aspectratio=L _(x) /L _(y) =∥k ₁ ∥/∥k ₂∥  (31)

Estimation of Pitch and Roll from Assumed Rectangle.

In practice the most common case is the camera capture of rectangulardocuments, such as sheets of paper of standard sizes, business cards,driver and other licenses, etc. Since the focal distance of the cameradoes not change, and since the knowledge of the yaw is irrelevant forthe discussed types of document image processing, it is necessary onlyto determine roll and pitch of the camera relative to the plane of thedocument in order to rectangularize the corresponding image of thedocument.

The idea of the algorithm is simply that one can calculate the objectcoordinates of the document corresponding to the tetragon found in thepicture (up to scale, rotation, and shift) for any relative pitch-rollcombination. This calculated tetragon in object coordinates ischaracterized by 90-degree angles when the correct values of pitch androll are used, and the deviation can be characterized by the sum ofsquares of the four angle differences. This criterion is useful becauseit is smooth and effectively penalizes individual large deviations.

A gradient descent procedure based on this criterion can find a goodpitch-roll pair in a matter of milliseconds. This has beenexperimentally verified for instances where the tetragon in the picturewas correctly determined. This approach uses yaw equal zero and anarbitrary fixed value of the distance to the object because changes inthese values only add an additional orthogonal transform of the objectcoordinates. The approach also uses the known focal distance of thecamera in the calculations of the coordinate transform, but if all fourcorners have been found and there are three independent angles, then thesame criterion and a slightly more complex gradient descent procedurecan be used to estimate the focal distance in addition to pitch androll—this may be useful for server-based processing, when incomingpictures may or may not have any information about what camera they weretaken with.

Interestingly, when the page detection is wrong, even the optimalpitch-roll pair leaves sizeable residual angle errors (of 1 degree ormore), or, at least, if the page was just cropped-in parallel to itself,the aspect ratio derived from the found object coordinates does notmatch the real one.

Additionally, it is possible to apply this algorithm even when alocation of one of the detected sides of the document is suspect ormissing entirely (e.g. that side of the document is partially orcompletely obstructed, not depicted, or is blurred beyond recognition,etc.). In order to accomplish the desired result it is useful to modifythe above defined criterion to use only two angles, for example thoseadjacent to the bottom side, in a gradient descent procedure. In thismanner, the algorithm may still be utilized to estimate pitch and rollfrom a picture tetragon with bogus and/or undetectable top-left andtop-right corners.

In one example, arbitrary points on the left and right sides closer tothe top of the image frame can be designated as top-left and top-rightcorners. The best estimated pitch-roll will create equally bogustop-left and top-right corners in the object coordinates, but thedocument will still be correctly rectangularized. The direction of amissing (e.g. top) side of the document can be reconstructed since itshould be substantially parallel to the opposite (e.g. bottom) side, andorthogonal to adjacent (e.g. left and/or right) side(s).

The remaining question is where to place the missing side in the contextof the image as a whole, and if the aspect ratio is known then theoffset of the missing side can be nicely estimated, and if not, then itcan be pushed to the edge of the frame, just not to lose any data. Thisvariation of the algorithm can resolve an important user case when thepicture contains only a part of the document along one of its sides, forexample, the bottom of an invoice containing a deposit slip. In asituation like this the bottom, left and right sides of the document canbe correctly determined and used to estimate pitch and roll; theseangles together with the focal distance can be used to rectangularizethe visible part of the document.

In more approaches, the foregoing techniques for addressing missing,obscured, etc. edges in the image data may additionally and/oralternatively employ a relaxed cropping and subsequent use ofconventional edge detection as described above with reference to FIG. 5.Of course, if the edge is completely missing from the image and/or videodata, then the relaxed cropping techniques may not be suitable to locatethe edges and projection as described above may be the sole suitablemechanism for estimating the location of edges. However, in the contextof the present disclosures, using internally represented content ratherthan corner or edge positions as key points allows projection of edgelocations in a broader range of applications, and in a more robustmanner than conventional edge detection.

As described herein, according to one embodiment a method 800 forreconstructing objects depicted in digital images based on internalfeatures of the object includes operations as depicted in FIG. 8. Aswill be understood by a person having ordinary skill in the art uponreading the present descriptions, the method 800 may be performed in anysuitable environment, including those depicted in FIGS. 1-2 and mayoperate on inputs and/or produce outputs as depicted in FIGS. 3A-5, invarious approaches.

As shown in FIG. 8, method 800 includes operation 802, in which aplurality of identifying features of the object are detected. Notably,the identifying features are located internally with respect to theobject, such that each identifying feature is, corresponds to, orrepresents a part of the object other than object edges, boundariesbetween the object and image background, or other equivalent transitionbetween the object and image background. In this manner, and accordingto various embodiments the presently disclosed inventive imagereconstruction techniques are based exclusively on the content of theobject, and/or are performed exclusive of traditional edge detection,border detection, or other similar conventional recognition techniquesknown in the art.

The method 800 also includes operation 804, where the digital image ofthe object is reconstructed within a three dimensional coordinate spacebased at least in part on some or all of the plurality of identifyingfeatures. In various embodiments, the portion of the image depicting theobject may be reconstructed, or the entire image may be reconstructed,based on identifying feature(s)

Of course, the method 800 may include any number of additional and/oralternative features as described herein in any suitable combination,permutation, selection thereof as would be appreciated by a skilledartisan as suitable for performing content-based object detection, uponreading the instant disclosures.

For instance, in one embodiment, method 800 may additionally oralternatively include reconstructing the digital image of the objectbased on transforming the object to represent dimensions of the objectas viewed from an angle normal to the object. As such, reconstructioneffectively corrects perspective distortions, skew, warping or“fishbowl” effects, and other artifacts common to images captured usingcameras and mobile devices.

Optionally, in one embodiment reconstructing the digital image of theobject is based on four of the plurality of identifying features, andemploys a four-point algorithm as described in further detail elsewhereherein. In such embodiments, preferably the four of the plurality ofidentifying features are randomly selected from among the plurality ofidentifying features. In some approaches, and as described in greaterdetail below, reconstruction may involve an iterative process wherebymultiple sets of four or more randomly selected identifying features areused to, e.g. iteratively, estimate transform parameters and reconstructthe digital image. Accordingly, reconstructing the digital image of theobject may be based at least in part on applying a four-point algorithmto at least some of the plurality of identifying features of the object,in certain aspects.

Reconstructing the digital image of the object may additionally and/oralternatively involve estimating a homography transform H. In oneapproach, estimating H comprises detecting one or more pointcorrespondences p_(i)↔P_(i)′ with p_(i)=(x_(i), y_(i), 1)^(T) asdiscussed above. Optionally, but preferably, each point correspondencep_(i)↔P_(i)′ corresponds to a position p_(i) of one of the plurality ofidentifying features of the object, and a respective position P_(i)′ ofa corresponding identifying feature of the reconstructed digital imageof the object. Estimating H may also include normalizing coordinates ofsome or all of the point correspondences.

As noted above, estimating the homography transform H may include aniterative process. In such embodiments, each iteration of the iterativeprocess preferably includes: randomly selecting four key points; using afour point algorithm to estimate an i^(th) homography transform H_(i)based on the four key points; and applying the estimated i^(th)homography transform H_(i) to a set of corresponding key points. Eachkey point corresponds to one of the plurality of identifying features ofthe object, and in some embodiments may be one of the plurality ofidentifying features of the object. The set of corresponding key pointspreferably is in the form of a plurality of point correspondences, eachpoint correspondence including: a key point other than the four randomlyselected key points; and a corresponding key point from a referenceimage corresponding to the digital image. The “other” key points alsocorrespond to one of the plurality of identifying features of theobject. Thus, each point correspondence includes two key points inpreferred embodiments: a key point from the test image and acorresponding key point from the reference image. The degree ofcorrespondence between point correspondences may reflect the fitness ofthe homography transform, in some approaches.

Thus, in some approaches method 800 may include evaluating fitness ofthe homography transform (or multiple homography transforms generated inmultiple iterations of the aforementioned process). The evaluation mayinclude determining one or more outlier key points from among each setof corresponding key points; identifying, from among all sets ofcorresponding key points, the set of corresponding key points having alowest number of outlier key points; defining a set of inlier key pointsfrom among the set of corresponding key points having the lowest numberof outlier key points; and estimating the homography transform H basedon the set of inlier key points. Preferably, the set of inlier keypoints exclude the outlier key points determined for the respective setof corresponding key points.

Furthermore, determining the one or more outlier key points from amongeach set of corresponding key points may involve: determining whethereach of the plurality of point correspondences fits a transformationmodel corresponding to the estimated i^(th) homography transform H_(i);and, for each of the plurality of point correspondences, either:designating the other key point of the point correspondence as anoutlier key point in response to determining the point correspondencedoes not fit the transformation model; or designating the other keypoint of the point correspondence as an inlier key point in response todetermining the point correspondence does fit the transformation model.

In several approaches, particularly preferred in the case of objectssuch as documents and especially standard documents such as forms,templates, identification documents, financial documents, medicaldocuments, insurance documents, etc. as would be understood by a skilledartisan upon reading the instant descriptions, the plurality ofidentifying features correspond to boilerplate content of the object. Invarious approaches, boilerplate content may include any type of suchcontent as described hereinabove.

Notably, employing reconstruction as set forth herein, particularly withrespect to method 800, carries the advantage of being able toreconstruct objects and/or images where at least one edge of the objectis either obscured or missing from the digital image. Thus, thepresently disclosed inventive concepts represent an improvement to imageprocessing machines and the image processing field since conventionalimage recognition and image processing/correction techniques are basedon detecting the edges of objects and making appropriate correctionsbased on characteristics of the object and/or object edges (e.g.location within image, dimensions such as particularly aspect ratio,curvature, length, etc.). In image data where edges are missing,obscured, or otherwise not represented at least in part, suchconventional techniques lack the requisite input information to performthe intended image processing/correction. It should be understood thatsimilar advantages are conveyed in the context of image recognition andmethod 700, which enables recognition of objects even where all edges ofthe object may be missing or obscured in the digital image data sincerecognition is based on features internal to the object.

In more embodiments, method 800 may include cropping the reconstructeddigital image of the object based at least in part on a projectedlocation of one or more edges of the object within the reconstructeddigital image. The projected location of the one or more edges of theobject is preferably based at least in part on an estimated homographytransform H.

In still more embodiments, method 800 may include classifying thereconstructed digital image of the object. As described in furtherdetail elsewhere herein, classification may operate as a type oforthogonal validation procedure or confidence measure for determiningwhether image recognition and/or reconstruction was performed correctlyby implementing the techniques described herein. In brief, if areconstructed image of an object is classified and results in adetermination that the object depicted in the reconstructed image is asame type of object represented in/by the reference image used toreconstruct the object, then it is likely the reconstruction wasperformed correctly, or at least optimally under the circumstances ofthe image data.

The foregoing descriptions of methods 700 and 800 should be understoodas provided by way of example to illustrate the inventive conceptsdisclosed herein, without limitation. In other approaches, thetechniques disclosed herein may be implemented as a system, e.g. aprocessor and logic configured to cause the processor to performoperations as set forth with respect to methods 700 and/or 800, as wellas a computer program product, e.g. a computer readable medium havingstored thereon computer readable program instructions configured tocause a processor, upon execution thereof, to perform operations as setforth with respect to methods 700 and/or 800. Any of the foregoingembodiments may be employed without departing from the scope of theinstant descriptions.

In addition, it should be understood that in various approaches it isadvantageous to combine features, operations, techniques, etc. disclosedindividually with respect to content based detection and content basedrecognition as described herein. Accordingly, the foregoing exemplaryembodiments and descriptions should be understood as modular, and may becombined in any suitable permutation, combination, selection, etc. aswould be understood by a person having ordinary skill in the art readingthe present disclosure. In particular, leveraging a four-point algorithmand estimating homography transforms to facilitate content-basedrecognition and content-based reconstruction of image data areespecially advantageous in preferred embodiments.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of an embodiment of the presentinvention should not be limited by any of the above-described exemplaryembodiments.

What is claimed is:
 1. A computer program product for reconstructing anobject depicted in a digital image, comprising a computer readablemedium having stored thereon computer readable program instructionsconfigured to cause a hardware processor, upon execution thereof, to:detect a plurality of identifying features of the object, wherein theplurality of identifying features are located internally with respect tothe object; and reconstruct the digital image of the object within athree dimensional coordinate space based at least in part on some or allof the plurality of identifying features.
 2. The computer programproduct as recited in claim 1, wherein reconstructing the digital imageof the object comprises transforming the object to represent dimensionsof the object as viewed from an angle normal to the object.
 3. Thecomputer program product as recited in claim 1, wherein reconstructingthe digital image of the object is based on four of the plurality ofidentifying features; and wherein the four of the plurality ofidentifying features are randomly selected from among the plurality ofidentifying features.
 4. The computer program product as recited inclaim 1, wherein reconstructing the digital image of the object is basedat least in part on applying a four-point algorithm to at least some ofthe plurality of identifying features.
 5. The computer program productas recited in claim 1, wherein reconstructing the digital image of theobject comprises estimating a homography transform H.
 6. The computerprogram product as recited in claim 5, wherein estimating H comprisesdetecting one or more point correspondences p_(i)↔P_(i)′ withp_(i)=(x_(i),y_(i),1)^(T).
 7. The computer program product as recited inclaim 6, wherein each point correspondence p_(i)↔P_(i)′ corresponds to aposition p_(i) of one of the plurality of identifying features of theobject, and a respective position P_(i)′ of a corresponding identifyingfeature of the reconstructed digital image of the object.
 8. Thecomputer program product as recited in claim 6, wherein estimating Hcomprises normalizing coordinates of some or all of the pointcorrespondences.
 9. The computer program product as recited in claim 5,wherein estimating the homography transform H comprises an iterativeprocess; wherein each iteration of the iterative process comprises:randomly selecting four key points, each key point corresponding to oneof the plurality of identifying features of the object; using a fourpoint algorithm to estimate an i^(th) homography transform H_(i) basedon the four key points; and applying the estimated i^(th) homographytransform H_(i) to a set of corresponding key points, wherein the set ofcorresponding key points comprises a plurality of point correspondences,each point correspondence including: an other key point corresponding toone of the plurality of identifying features of the object; and acorresponding key point from a reference image corresponding to thedigital image.
 10. The computer program product as recited in claim 9,comprising program instructions configured to cause the processor to:determine one or more outlier key points from among each set ofcorresponding key points; identify, from among each set of correspondingkey points, the set of corresponding key points having a lowest numberof outlier key points; define a set of inlier key points from among theset of corresponding key points having the lowest number of outlier keypoints, the set of inlier key points excluding the outlier key pointsdetermined from among the set of corresponding key points having thelowest number of outlier key points; and estimate the homographytransform H based on the set of inlier key points.
 11. The computerprogram product as recited in claim 10, wherein determining the one ormore outlier key points from among each set of corresponding key pointscomprises: determining whether each of the plurality of pointcorrespondences fits a transformation model corresponding to theestimated i^(th) homography transform H_(i); and, for each of theplurality of point correspondences, either: designating the other keypoint of the point correspondence as an outlier key point in response todetermining the point correspondence does not fit the transformationmodel; or designating the other key point of the point correspondence asan inlier key point in response to determining the point correspondencedoes fit the transformation model.
 12. The computer program product asrecited in claim 1, wherein the plurality of identifying featurescorrespond to boilerplate content of the object.
 13. The computerprogram product as recited in claim 1, wherein at least one edge of theobject is either obscured or missing from the digital image.
 14. Thecomputer program product as recited in claim 1, comprising cropping thereconstructed digital image of the object based at least in part on aprojected location of one or more edges of the object within thereconstructed digital image.
 15. The computer program product as recitedin claim 14, wherein the projected location of the one or more edges ofthe object is based at least in part on an estimated homographytransform H.
 16. The computer program product as recited in claim 1,comprising classifying the reconstructed digital image of the object.17. A computer program product for detecting a financial instrumentdepicted in a digital image, comprising a computer readable mediumhaving stored thereon computer readable program instructions configuredto cause a hardware processor, upon execution thereof, to: detect, usingthe hardware processor, a plurality of identifying features of thefinancial instrument, wherein the plurality of identifying features arelocated internally with respect to the financial instrument; project,using the hardware processor, a location of one or more edges of thefinancial instrument based at least in part on the plurality ofidentifying features; and output, at least in part based on theprojected location of the one or more edges of the financial instrumentto a display of a computer, and/or a memory; and wherein the financialinstrument is a check.
 18. The computer program product as recited inclaim 17, wherein the plurality of identifying features correspond toboilerplate content of the check.
 19. The computer program product asrecited in claim 17, wherein the plurality of identifying features areselected from the group consisting of: a “MEMO” region of the check, apayee designator, a bank name, a bank address, a payor name, a payoraddress, one or more MICR characters, an endorsement line, a dollarsign, a padlock icon, a “PAY TO THE ORDER OF” region of the check, and apayment amount region of the check.
 20. A computer program product fordetecting an object depicted in a digital image, comprising a computerreadable medium having stored thereon computer readable programinstructions configured to cause a hardware processor, upon executionthereof, to: detect, using the hardware processor, a plurality ofidentifying features of the object, wherein the plurality of identifyingfeatures are located internally with respect to the object; project,using the hardware processor, a location of one or more edges of theobject based at least in part on the plurality of identifying features;and output the projected location of the one or more edges of the objectto a display of a computer, and/or a memory.